This is not possible as a standard. However you can create custom lightning web component with checkbox. You can refer to How to use custom LWC lightning component in lightning-datatable for detailed explanation on how to use custom LWC component in a column. With this solution you will not get select all checkbox.

Custom component HTML:

This is a very minor work-around compared to building datatable from scratch

Coming in late here OP but this is possible with standard functionality in the platform.

Include the maxRowSelection attribute within your aura lightning datatable definition defaulted to 1, pseudo:

this will remove the selectAll checkbox and transform the checkboxes to radio buttons as specified here: https://developer.salesforce.com/docs/component-library/bundle/lightning:datatable/specification.